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Abstract 


Background / Context: 

Description of prior research and its intellectual context. 


Outcome reporting bias refers to the practice of omitting from primary studies outcomes that 
were actually collected. When primary studies do not report on all the outcomes assessed, we 
have an incomplete understanding of a phenomenon that may be compounded when the study is 
included in a systematic review of research. Outcome reporting bias is well-documented in the 
medical literature (see, for example, Vedula et ah, 2009 ; Chan et ah, 2004) through the 
comparison of research protocols and unpublished reports of research with their published 
version. In many of these studies, statistical significance was related to the likelihood that an 
outcome was reported. In education, a recent paper by Pigott et al. (2013) demonstrates the 
presence of outcome reporting bias in educational research. Through a comparison of 
dissertations with their published versions, Pigott et al. found that outcomes failing to produce a 
statistically significant result in a dissertation were 30% more likely to be omitted from a 
published version of the dissertation when compared to those outcomes that were statistically 
significant. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The purpose of the proposed paper is to extend the work done by Pigott et al. (2013) on 
outcome-reporting bias in educational research to examine other potential correlates for 
incomplete reporting. For example, one set of potential reasons for outcome reporting bias 
relates to the characteristics of the outcome such as whether the outcome was considered a 
primary or a secondary focus of an intervention, whether the outcome was a negative or 
potentially harmful effect of the intervention, and whether the measurement strategy for the 
outcome was reliable and valid. Other potential causes of outcome reporting bias relate to the 
purpose of the analysis. Researchers may omit mention of the results of tests of baseline 
equivalence in a quasi-experimental study if those results are unfavorable. Researchers may also 
use different analysis strategies in the published version of a dissertation if they are able to reach 
statistical significance. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 

The significance of this study is to provide the education research community with a deeper 
understanding of outcome-reporting bias and the degree to which it has been present in the field 
of education. This paper will build upon the work of Pigott et al. (2013)to explore what other 
factors might be related to outcome-reporting bias. Outcome reporting bias impacts the 
completeness and accuracy of research findings, and leads to an incomplete understanding of a 
phenomenon. Using biased results in a systematic review can compound this problem, and lead 
researchers to erroneous conclusions that could be harmful to students. Understanding the 
potential reasons for incomplete reporting can help to increase the quality of education research 
and the confidence in its findings. 
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Research Design: 

Description of the research design. 


The original work by Pigott et al. (2013) examined a limited set of correlates of outcome 
reporting bias. In addition to statistical significance, Pigott et al. looked at time to publication, 
sample size of the study, number of statistical tests conducted and type of outcome measure as 
potentially related to the likelihood of an outcome appearing in the published version. No 
evidence of a relationship was found between these other factors and whether an outcome was 
reported in a published version of the research. In this study, we are going back to the original 
79 dissertation-published paper pairs to code other potential reasons for outcome reporting bias. 

One set of codes relates to characteristics of the outcomes collected. Vedula et al. (2009) found 
that many outcomes reported as primary in the unpublished versions were considered secondary 
in the published version when the result was harmful or inconsistent with the researchers’ initial 
hypothesis. In the original Pigott et al. data, we are coding for each dissertation outcome the 
following: the original hypothesis (i.e., the expected direction, negative or positive, of the impact 
of the intervention on the outcome), whether the results for that outcome conformed with the 
original hypothesis, and whether the outcome was considered primary or secondary to the 
research question in the study. Another set of codes is related to work by loannidis (2005) on 
false research findings. loannidis suggests that in fields where there is not agreement on the 
measurement of an outcome, or where there is flexibility in the types of analyses used, there is a 
greater potential for selective reporting. Given loannidis’ work, we are coding the following: 
whether the outcome was measured using an instrument with demonstrated reliability and 
validity, whether the outcome was a standardized instrument widely used in the field or whether 
it was researcher-constructed, and whether the analysis strategy changed from the dissertation to 
the published version. For this last code, we will need to go back to the originally identified set 
of dissertation-published study pairs since we originally excluded pairs that did not use the same 
analysis strategy from the dissertation to the published version. 

In addition, we will conduct analyses on other ancillary hypothesis tests included in the 
dissertation pairs that were not used in the original paper. For example, tests of baseline 
equivalence, normality, and homogeneity tests were excluded from the original analysis, but will 
be included here to see if outcome reporting bias of these ancillary tests exists and is related to 
any of our coded variables. 

Coding will be conducted by the first two authors independently for all pairs of dissertations- 
published papers. The analysis strategy will follow the original paper, using the Mantel- 
Haenszel mean odds ratio (Shadish& Haddock, 2009) to examine how the odds of an outcome 
appearing in the published version is related to the various codes described above. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 


The data sources will include the 79 dissertation-published paper pairs collected by Pigottet al. 
(2013). These 79 pairs examine an educational intervention conducted with K-12 students using 
either an experimental or quasi-experimental design. The sample was limited to intervention 
studies to simplify the identification of targeted outcomes for an intervention as opposed to 
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outcomes in observational studies. The sample of dissertations were identified by searching the 
ProQuest Dissertation and Theses Database between the years 2001 and 2005 using the keyword, 
Education, and limiting results to Ph.D and Ed.D. degrees. A total of 9,530 dissertations were 
identified. Of these dissertations, 621 reported on an educational intervention for students in 
PreK-12. Within the 79 dissertations, 1,599 treatment outcomes were reported. 

For this update, we are expanding the coding of the dissertation-published paper pairs using the 
variables described above. We are also checking for published versions of the original 621 
dissertations that may have appeared since the search ended in May 2011. 

Findings / Results: 

Description of the main findings with specific details. 


Coding of the original pairs is currently on-going. We are also searching for any additional 
matches to identified dissertations since 2011. Analysis will be complete by December 2014. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


Coding of the original pairs is currently on-going. We are also searching for any additional 
matches to identified dissertations completed since 2011. Analysis will be complete by 
December 2014. 
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